This disclosure relates to means for analyzing specialized data and index files of KEYEDIOII Data Management System Software for more efficient usage.
The operational factors of the present method involve the KEYEDIOII software. This software is the Unisys Indexed Sequential Access Method (ISAM) software for COBOL 74 and for Report Program Generator (RPG) programming languages.
KEYEDIOII is a data management system (DMS II) based product developed to take advantage of certain of the DMS II features in order to achieve better performance for KEYEDIO users.
Table A-1 is an example of the Block 0 Format used for data files. The KEYEDIOII data file is declared with MAXRECSIZE equal thirty words and Block Size equal thirty words. There are logical blocks which exist within the file. The logical block size of a file is a minimum of 120 words or the user-declared block size translated into words. Records are always one word greater than the size of the user-declared record translated into words. When an existing KEYEDIOII file is opened, the user file declaration MAXRECSIZE is corrected to the original user MAXRECSIZE declaration instead of the thirty-word physical size.
Because logical block 0 of the data file is always reserved, the user record 0 is actually located in the first record of block 1. The last word of each record (hidden word) contains either the user logical-record number (starting with 0) or else all bits turned on, which indicates a deleted record.
There is a Block 0 Format for index Files: Table B-1 is an illustration of the Block 0 Format for the index Files. The physical MAXRECSIZE and the Block Size for the Index Files are thirty words. Logical Block Size is currently defined to be 900 words. The data files have the table logical Block Size stored in a field called the TABSIZE field of block 0.
KEYEDIOII is an Indexed Sequential Access Method (ISAM) software for COBOL 74 and Report Program Generator (RPG) programming languages.
KEYEDIOII support is provided by an initialized system library. The function name is KEYEDIOIISUPPORT and a library code file is named SYSTEM/KEYEDIOII. A conversion is required to change KEYEDIO files (earlier version) to KEYEDIOII files, however, both types of these files can coexist. When a KEYEDIO file is accessed, it connects to the KEYEDIOSUPPORT function; when a KEYEDIOII file is accessed, it connects to the KEYEDIOII support function. In either case, the compiler to the support-library interface is identical, so that user programs need not be recompiled to create an access KEYEDIOII file.
KEYEDIOII Software Files
These software files consist of four code files and three symbol files, which will be designated with the letter xe2x80x9ccxe2x80x9d for code and xe2x80x9csxe2x80x9d for symbol.
The code files are: (c1) SYSTEM/KEYEDIOII; (c2) SYSTEM/KEYEDIOII/FORMLID; (c3) SYSTEM/KEYEDIOII/UTILITY; (c4) KIOII/FORMS/TRDESCRITTION.
The symbol files are designated as follows: (s1) SYMBOL/KEYEDIOII/FORMLID; (s2) SCREENDESIGN/KIOII/FORMS; (s3) SYMBOL/KEYEDIOII/UTILITY.
Installing KEYEDIOII
The first step in installation is to select a pack on which to load the SYSTEM/KEYEDIOII code file and then to initialize the KEYEDIOIISUPPORT function.
On first use of KEYEDIOII, the system creates a parameters file with default values on the same pack as the one with the SYSTEM/KEYEDIOII code file. The title of the parameters file is KEYEDIOII/CONTROL. This file contains run-time parameters for KEYEDIOII. These parameters affect the core usage, the audit check pointing and the ability to print run-time statistics.
The Master Control Program (MCP) initiates the KEYEDIOII libraries when the first KEYEDIOII file is opened.
KEYEDIOII Libraries
When the first KEYEDIOII file is created or accessed, then two KEYEDIOII library stacks are run. These are displays in the LIBRARY entries, as indicated below in Table L1.
The user column shows a number of open KEYEDIOII files. The task 1235, which is the freeze control library, accepts AX (accept) messages to change the KEYEDIOII parameters previously mentioned.
To bring down the KEYEDIOII libraries, it is necessary to enter the ODT THAW command with either one of the library stacks. When the user count is zero (after all user programs have closed their KEYEDIOII files), the KEYEDIOII library tasks will end. The KEYEDIOII/LIBRARY checks to see if the THAW command has been executed. It does so once every minute or whenever the libraries EXCEPTION event is caused. Then the operator enters the HI (to cause the EXCEPTION event) ODD command after the THAW command to avoid waiting for up to one minute for the library to end.
In addition to the library stacks, SYSTEM/KEYEDIOII creates as many as four other stacks. Each of these stacks can accommodate up to 250 structures; two structures are allocated for each data file and a single structure for each index file. These stacks are active entries named KEYEDIOII/STRUCTURES nn, where nn is 00 for the first stack thru 03 for the fourth stack. These stacks become active one at a time when needed and remain active as long as the libraries are frozen. These stacks contain control information for each of the currently open KEYEDIOII files.
System Operations
There are certain file naming conventions and defaults used by KEYEDIOII, plus the creation and maintenance of file indexes and what is called COPY utility.
A KEYEDIOII file consists of one or more physical files. There is a data file containing the KEYEDIOII records and also an index file for each key or group of keys. The title of the data file is declared in the user program that creates, or reads, or updates the file.
By default, the index files have the same name as that of the data file, but /KEYnn is appended to the end of the file name. The variable nn in the index file name is the key number assigned at creation time. By default, the system creates index files on the same pack where the data file resides. As an example, if a user creates a program with a KEYEDIOII file which is entitled as a user-code (UC) ABC ON PACK that has three keys, then the system will create the following types of physical files:
(i) (UC) ABC ON PACK;
(ii) (UC) ABC/KEY01 ON PACK;
(iii) (UC) ABC/KEY02 ON PACK;
(iv) (UC) ABC/KEY03 ON PACK;
The system assigns the key number at the time that it creates the index. The key number remains permanently assigned as long as the index exists. If a user should delete (UC) ABC/KEY02, the other indices will still keep their same titles. If a programmer adds a new index, the system assigns an index number 2 to the new index because now xe2x80x9c2xe2x80x9d is the first available unassigned index number.
There are certain advantages for having separate physical files for the indexes. These include (a) the indexes can be relocated on different packs in order to balance the I/O paths; (b) indexes can be reconstructed from the data file if they become corrupted.
Relocating Index Files on Different Files
Block 0 of the data file (Table A-1) contains a directory of index file titles and pack locations. If a directory is empty, all the defaults are used. If a user wants to relocate index files on different packs, the user can use one of the following two methods to relocate the index files:
Method 1: Use the LIBRARY/MAINTENANCE function to copy the index file to the new pack and then remove the index file from the old pack. Then on a next execution of a user program that accesses the data file, the user will enter the ?FA command for the appropriate index file and pack name.
When the file is found, KEYEDIOII updates a directory in Block 0 of the data file (Table A-1) so that the index is found in the new pack the next time that the user program executes.
Method 2: Using the menu-driven function provided by the SYSTEM/KEYEDIOII/UTILITY program to relocate the indexes to different packs.
Utility Screens
The KEYEDIOII utility provides functions for adding, deleting, recreating, and moving index files and copying files. These KEYEDIOII utility functions are provided by the program SYSTEM/KEYEDIOII/UTILITY. The source file is named SYMBOL/KEYEDIOII/UTILITY. A user can call the utility as a library to programmatically invoke on-line functions.
Function Selection on Main Menu
The first screen that appears after a user invokes the utility is a menu that allows a user to select the activity the user wants to perform. The user can add, delete, recreate, or move an index for a KEYEDIOII file, or the user can copy various kinds of files. The user must enter a letter in the Function field and must name a source file for all options, except the final exit option of Quitting.
Accessing Files with Group Keys
In KEYEDIO, a one-to-one correspondence exists between the keys and the indexes, that is to say a single key field in the record provides an access to the KEYEDIO file. In KEYEDIOII, there is file access allowed by means of a group key. A group key is composed of multiple non-contiguous key fields in the records that are concatenated to form a group key for accessing files.
Locking Records
Locking records is an option in KEYEDIOII. By default, for compatibility with KEYEDIO, the records are not locked. Therefore, between the time when a record is being read and the time it is written, the record can be changed by another user of the file.
It is possible to enable record locking by assigning file attribute RECORDLEVELLOCK the value TRUE before opening a new KEYEDIOII file. After a file has been created with RECORDLEVELLOCK equal TRUE, the attribute becomes permanent. If a RECORDLEVELLOCK is enabled, the system enforces record locking when the file is opened for updating, that is for FILEUSE equals VALUE (IO). The system locks the record each time that the file is read. If the file is opened for enquiry, the system does not lock the records being read.
Subsequently the locked record can be freed by performing a certain series of tasks.
Performance and Usage
One factor is that of calculating the buffer size. The size of the buffer pool depends on the value of the ALLOWEDCORE parameter, which is controlled by the AX (accept) ODT command, which is a significant performance aspect of KEYEDIOII. The ALLOWEDCORE parameter represents the total buffer pool core available for all users of KEYEDIOII files. This parameter represents both a minimum and a maximum: it represents a minimum in that KEYEDIOII brings in additional blocks until ALLOWEDCORE is reached; it represents a maximum in that once the ALLOWEDCORE is reached, then the KEYEDIOII will tend to reuse the buffers.
If only one KEYEDIOII file is open, then KEYEDIOII might be able to bring the entire file into core, which would make access very fast. If more than one KEYEDIOII file is open, the ALLOWEDCORE is shared among the files so not as much of any one given file will fit in the ALLOWEDCORE. The buffers are reused, based on an algorithm that determines the least recently used buffer. Thus frequently access blocks tend to remain in core, and the less frequently used blocks are flushed out for new blocks.
The default value for ALLOWEDCORE is 50,000. As an example, one user site generally runs with 50 KEYEDIOII files open, and those 50 files have 200 users. That site runs with ALLOWEDCORE set at 200,000 words. The AX STATUS command will display the amount of ALLOWEDCORE actually in use. A quickly increasing FORCED OVERLAY count would indicate that the ALLOWEDCORE value is probably too small and that thrashing is occurring in the KEYEDIOII buffer pool.
Block Size Calculation
In general, it is useful to have small blocking factors for random accessing and large blocking factors for sequential accessing of files. A very important feature of KEYEDIOII is the ability to reblock when sequential accessing is detected. When the system detects physically sequential access (as opposed to KEYEDIO KEY sequential), then the KEYEDIOII automatically starts reading larger blocks. This process is called xe2x80x9creblocking xe2x80x9d. Blocks up to 40 times the original block size, or approximately 3,000 words, whichever is smaller, are read into memory when physically sequential accessing is detected.
Generally, user-declared block sizes need not be large, because KEYEDIOII automatically adjusts them upward on sequential access. Block sizes normally should probably not exceed 1,000 words so that transfer time does not become a significant item on random I/O operations. A block size of about 500 words could be considered adequate.
As noted, when reblocking is in force, blocks are around 3,000 words in size. The only reason for declaring larger blocks is that population of the file is small enough that most of the file fits in ALLOWED CORE, and that most of the file is touched during a particular job. It is faster to search a small number of large blocks, than to search a large number of small blocks to see whether or not a particular block is in CORE.
Although the block sizes can be controlled by the users, KEYEDIOII still might override the user specification in certain ways, as follows: (a) the system adds one word of hidden data to each KEYEDIOII record so that KEYEDIOII attempts to re-adjust the block size in order to minimize wasted space; (b) block 0 of every KEYEDIOII file is reserved for control information, and KEYEDIOII reserves a minimum of 120 words for this information. (Thus, the user record xe2x80x9c0xe2x80x9d is actually the first record of block 1). Thus, the block sizes are always a minimum of 120 words.
Accessing Files with Relative Keys
When a KEYEDIO file is declared as xe2x80x9cIndexed,xe2x80x9d the file can be accessed only through a user-declared index key (for example, an alpha field in the record). If the file is declared as INDEXNOTRESTRICTED, the file can be accessed either by the user-declared index key or a xe2x80x9crelative keyxe2x80x9d (a numeric record number relative to record 0 in the file). Relative key accessing treats the file as though it were a flat file. In ALGOL, a Read statement of the form Read (F[RKEY], 30,A) means to read the record at the relative record RKEY of the file. With the earlier KEYEDIO files, there was some performance cost associated with files that are INDEXNOTRESTRICTED. Relative keys are put into a hidden index on the file similar to a user-declared index. Thus, adding a record to a file involves inserting a key into two indexsequential tables.
However, with KEYEDIOII, the ability to perform relative key accessing is essentially free. All KEYEDIOII files, even those declared as Indexed, are compatible with the relative key accessing programs, because a form of Data Management System II (DMSII) allows Direct data sets to be used for the KEYEDIOII files.
Unlike the prior KEYEDIO, the relative key order always corresponds to the physical record order in the file. Accessing in physical order is guaranteed in the KEYEDIO only when the file is originally created in user-key order and no keys have been added or deleted (since reused space causes new relative keys to be put into the middle of the file). Even then, however, cable blocks are mixed in with the data blocks.
As previously noted in xe2x80x9ccalculating the block size,xe2x80x9d the reblocking occurs with substantial performance gains when the system detects physically sequential access. In certain situations where the user wants to read all the records in the file in no particular order, then relative key accessing can be used to obtain performance gain.
When a file is opened, the index key of reference becomes the primary key (NRPG, it is the first key declared). When a file is read sequentially, the system returns records in the key of reference order. The key of reference can be altered by including a start statement in the program.
In ALGOL, the first sequence of code returns records in primary key order, which would probably not be in the physical order. There is a second sequence of code which returns records in physical order and will initiate reblocking.
In COBOL, relative key accessing is controlled by the Select clause. When the system reads the file, it returns records in physical sequential order. The system them updates RKEY to reflect what relative record is returned each time that the system sequentially reads the file. Deleted records are skipped. There are certain types of Select clauses in programs which allow random access to a file by using a relative key. RKEY is equal to the relative key value specified before the file is read.
In RPG (Report Program Generator), the file is declared as though it were a flat file, that is to say as a non-KEYEDIOII file that is accessed like any other disk file.
Running the System
The system writes records to the KEYEDIOII audit files under the following situations:
(a) Records are added to an existing KEYEDIOII file.
(b) Records are deleted from an existing KEYEDIOII file.
(c) Key changes are made to records of an existing KEYEDIOII file.
(d) A block that has been modified by some other program is encountered and the block has not yet been written back to disk.
The system does not write records to the KEYEDIOII audit files under the following situations:
(i) When a new file is created.
(ii) When a record of an existing file that does not affect the key field is updated. The data block is written back directly instead of being written to the audit file.
(iii) When a KEYEDIOII file has the file attribute NORECOVERY equals TRUE. This situation would normally be used by a batch program, where the file has been dumped prior to the execution. If a failure occurs, the system reloads the file and reruns the program.
Performance can be significantly improved by running the system without auditing. The system runs without auditing when NORECOVERY equals TRUE. Records are not written to the audit file, and multiple updates to the same block can take place and then they are written only once when the buffer is pushed out of ALLOWEDCORE or the file is closed.
If a file is updated when NORECOVERY equals TRUE, and if the Halt Load occurs before the file is closed, the file becomes unusable and must be reloaded after the Halt Load. Thus under the proper circumstances, NORECOVERY can be used to improve performance.
The present invention enables the user to analyze the condition of data files and index files of the specialized KEYEDIOII files so that subsequently the user would be able to initiate operations to restructure the KEYEDIOII files in a more efficient arrangement enabling faster access to the files and faster reading and writing of files.
The present invention provides a method involving means for analyzing data files in order to recognize when they are structured in an inefficient arrangement. This is done both for data file arrays and also for index file arrays.
The present invention relates to methods for analyzing specialized files designated as KEYEDIOII. The method provides a highly user-iterative tool which allows the user to analyze only those data files and index files that are of the specialized KEYEDIOII type.
If the file involved is of the KEYEDIOII type data file, then the methodology tool will analyze all blocks of the file. This makes it possible then to recognize when the files are structured in an inefficient arrangement.
Similarly, if the accessed file is of the index file type of KEYEDIOII, then the methodology tool scans through all the tables and analyzes each table.
Finally, the method tool outputs the analytical information onto a suitable output medium such as a screen or a printer. The output information displayed could be that of the following items:
(a) Name of the KEYEDIOII file;
(b) The file attributes, which include such items as:
(i) File size;
(ii) File location;
(iii) Creation/modification with time/date information;
(iv) Accessibility attributes.
(i) Amount of data filled as a percentage of the file size;
(ii) The depth of table level;
(iii)An analysis of block levels such as the blocks as a percentage of the entire actual entries.